Set-up

Packages

pacman::p_load(tidyverse,
               here,
               janitor)

Data

• we’ll use the languageR_english.csv dataset (in daten folder)

df_eng <- read_csv(here("daten", "languageR_english.csv")) |> 
1  clean_names() |>
2  arrange(word) |>
3  rename(
4    rt_lexdec = r_tlexdec,
5    rt_naming = r_tnaming
  ) |> 
6  select(age_subject, word, word_category, rt_lexdec, rt_naming)

Tidy workflow

• Abbildung 1 shows an overview of the typical data science process
  • import our data, tidy it, then work through a cycle of transforming, visualising, and modelling before finally communicating your findings
• today we’ll learn about the tidying step

Abbildung 1: A model of the data science process from Wickham et al. (2023) (all rights reserved)

Tidy data

• the same data can be representing multiple ways
• we’re going to look at 3 tables that present the exact same data in different formats
• the tables show the same values of four variables:
  • country
  • year
  • population
  • and number of tuberculosis cases
• Each dataset organises the values differently
• As we look at each table, take a moment to consider how easy it is to understand the values and relationships between the variables

Rules for tidy data

• You likely found Tabelle 1 easiest to read
  • it follows the three rules for tidy data (visualised in Abbildung 2):

1. Each variable is a column, each column is a variable
2. Each observation is a row, each row is an observation
3. Each value is a cell, each cell is a single value

Abbildung 2: Image source: Wickham et al. (2023) (all rights reserved)

Why tidy data?

“Happy families are all alike; every unhappy family is unhappy in its own way.”

— Leo Tolstoy

“Tidy datasets are all alike, but every untidy dataset is untidy in its own way.”

— Hadley Wickham

There are two main advantages to working with tidy data:

1. working with a consistent data structure allows us to adopt conventions
   • tidy data is the generally agreed upon data structure
   • conventions/tools are built on the assumption of this structure
2. R’s vectorised nature can shine
   • most built-in R funtions work with vector values (and columns are essentially vectors)
   • all packages in the tidyverse are designed to work with tidy data (e.g., ggplot2 and dplyr)

Data tidying

• transforming wide data to long data and long data to wide data (among other steps)
  • outcome: tidy data (usually)

Data tidying with the tidyverse

• tidyr package (from the tidyverse) has two useful functions for transposing our data:

  • pivot_longer(): make wide data longer
  • pivot_wider(): make long data wider

Abbildung 3: die berühmteste Verwendung des Wortes Pivot (zumindest für Millenials) (Friends)

Wide versus long data

• we often need to convert between wide and long data formats to do different types of summaries or visualisation

• wide data: all of the observations about one thing are in the same row

  • is usually not tidy

• long data: each observation is on a separate row

  • is usually tidy

• let’s start with the most typical case: turning wide data into long data

pivot_longer()

• in the languageR_english.csv dataset (df_eng)
  • we have 4568 observations (rows)
  • we have 7 variables (columns)
  • the column age_subject tells us whether an observation is from a participant in the ‘old’ or ‘young’ age group
  • the columns word, length_in_letters, written_frequency, and word_category describe properties of the stimulus for a given observation (i.e., the word)
  • the column rt_lexdec contains the response time for a lexical decision task
  • the column rt_naming contains the response time for a word naming task

head(df_eng)

• is this data in Tabelle 4 tidy?
• is this data too wide or too long?
• how might we make this data longer?

Our goal

• we want to produce Abbildung 4

Abbildung 4: Our plot to be reproduced

• the two continuous variables rt_lexdec and rt_naming appear in facets
  • facet_wrap() takes a categorical variable as its argument, and produces a facet of each category
• so we need a categorical variable which contains the levels lexdec and naming
  • and a continuous variable that contains the corresponding response time

pivot_longer()

• tidyr function pivot_longer() converts a wide data table to a longer format
  • converts the names of specified columns into the values of new a categorical column
  • and combines the values of those columns into a new column

df_eng_long <- 
  df_eng %>% 
  pivot_longer(
    cols = starts_with("rt_"), 
    names_to = "response", 
    values_to = "time"
  )

• let’s compare the observations for the words ace and act in
  • df_eng (Tabelle 5)
  • df_eng_longer (Tabelle 6)

Plotting our tidy data

• now let’s try to create our plot:
  • age_subject on the x-axis
  • time on the y-axis
  • response categories in facets

Abbildung 5: Response times per age group for the lexical decision task vs. naming task

pivot_wider()

• it’s more common to want to lengthen your data (take columns and turn their values into new rows)
  • but sometimes we also want to widen our data (take rows and turn their values into new columns)
• tidyr function pivot_wider() make datasets wider by increasing columns and reducing rows
  • this helps when one observation is spread across multiple rows
• let’s try to make df_eng wider
  • for example, we could have a single row per word
    • with a single variable for the young subject’s response and the old subject’s response

pivot_wider()

• pivot wider takes similar arguments to pivot_longer(), with some slight differences:

  • id_cols (optional): identifying columns (which columns uniquely identify each observation?)
  • names_from: what should we call the new column containing the previous column names (must be a categorical variable)?
  • names_prefix (optional): prefix for the new column names (optional)
  • values_from: new column values

• let’s create two new variables that take their names from age_subject, and their values from rt_lexdec. The result should look like Tabelle 8.

df_eng_wide <-
  df_eng %>%  
  select(-rt_naming) |> 
  pivot_wider(
1    names_from = age_subject,
2    values_from = rt_lexdec,
3    names_prefix = "lexdec_"
  )

• let’s compare the observations for the words ace and act in
  • df_eng (Tabelle 5)
  • df_eng_longer (Tabelle 6)

• again, we haven’t changed any data or observation values, we’ve just simply re-structured the organisation of the data points

Homework

We’ll stick with the df_eng dataset for these tasks.

1. Using pivot_wider to create rt_naming new variables: naming_old and naming_young which containg the naming response times for old and young participants, respectively. Hint: you will need to remove rt_lexdec. The resulting data frame should have 2284 observations and 6 variables.

2. Recreate Abbildung 6. Hint: you will need pivot_wider().

Abbildung 6: Scatterplot of lexical decision task response times per word for old versus young participants

3. Why do we need out df_eng_wide dataset to produce Abbildung 6? In other words, why is df_eng_wide the appropriate structure but not df_eng_long for such a scatterplot?

4. Using df_eng_long and the summarise() function we saw in the last section, reproduce the summary below:

# A tibble: 2 × 3
  response   mean    sd
  <chr>     <dbl> <dbl>
1 rt_lexdec  708.  115.
2 rt_naming  566.  101.

Hint: do you need to remove NAs (we saw how to do this in the last section)?